Fusion of Model-free Reinforcement Learning with Microgrid Control: Review and Vision

Challenges and opportunities coexist in microgrids as a result of emerging large-scale distributed energy resources (DERs) and advanced control techniques. In this paper, a comprehensive review of microgrid control is presented with its fusion of model-free reinforcement learning (MFRL). A high-level research map of microgrid control is developed from six distinct perspectives, followed by bottom-level modularized control blocks illustrating the configurations of grid-following (GFL) and grid-forming (GFM) inverters. Then, mainstream MFRL algorithms are introduced with an explanation of how MFRL can be integrated into the existing control framework. Next, the application guideline of MFRL is summarized with a discussion of three fusing approaches, i.e., model identification and parameter tuning, supplementary signal generation, and controller substitution, with the existing control framework. Finally, the fundamental challenges associated with adopting MFRL in microgrid control and corresponding insights for addressing these concerns are fully discussed.Comment: 14 pages, 4 figures, published on IEEE Transaction on Smart Grid 2022 Nov 15. See: https://ieeexplore-ieee-org.utk.idm.oclc.org/stamp/stamp.jsp?arnumber=995140

Virtual Inertia Scheduling (VIS) for Real-Time Economic Dispatch of IBRs-Penetrated Power Systems

A New Concept Called Virtual Inertia Scheduling (VIS) is Proposed to Efficiently Handle the Increasing Penetration of Inverter-Based Resources (IBRs) in Power Systems. VIS is an Inertia Management Framework that Targets Security-Constrained and Economy-Oriented Inertia Scheduling and Generation Dispatch with a Large Scale of Renewable Generations. Specifically, It Determines the Appropriate Power Setting Points and Reserved Capacities of Synchronous Generators and IBRs, as Well as the Control Modes and Control Parameters of IBRs to Provide Secure and Cost-Effective Inertia Support. First, a Uniform System Model is Employed to Quantify the Frequency Dynamics of the IBRs-Penetrated Power Systems after Disturbances. Leveraging This Model, the s-Domain and Time-Domain Analytical Responses of IBRs with Inertia Support Capability Are Derived. Then, VIS-Based Real-Time Economic Dispatch (VIS-RTED) is Formulated to Minimize Generation and Reserve Costs, with Full Consideration of Dynamic Frequency Constraints and Derived Inertia Support Reserve Constraints. the Virtual Inertia and Damping of IBRs Are Formulated as Decision Variables. a Deep Learning-Assisted Linearization Approach is Further Employed to Address the Non-Linearity of Dynamic Constraints. Finally, VIS-RTED is Demonstrated on a Two-Machine System and a Modified IEEE 39-Bus System. a Full-Order Time-Domain Simulation is Performed to Verify the Scheduling Results and Ensure their Feasibility

Improving Virtual Synchronous Generator Control in Microgrids using Fuzzy Logic Control

Virtual synchronous generators (VSG) are designed to mimic the inertia and damping characteristics of synchronous generators (SG), which can improve the frequency response of a microgrid. Unlike synchronous generators whose inertia and damping are restricted by the physical characteristics of the SG, VSG parameters can be more flexibly controlled to adapt to different disturbances. This paper therefore proposes a fuzzy logic controller designed to adaptively set the parameters of the VSG during a frequency event to ensure an improved frequency nadir and rate of change of frequency (ROCOF) response. The proposed control method is implemented and tested on the power inverter for the battery energy storage system of the Banshee Microgrid Feeder 2 test case system using MATLAB/SIMULINK. The effectiveness of the adaptive control scheme is validated by comparing its performance with a constant parameter VSG, a virtual inertia only fuzzy controller, and an inertial-less inverter control

Decentralized and Coordinated Vf Control for Islanded Microgrids Considering DER Inadequacy and Demand Control

This paper proposes a decentralized and coordinated voltage and frequency (Vf) control framework for islanded microgrids, with full consideration of the limited capacity of distributed energy resources (DERs) and Vf dependent load. First, the concept of DER inadequacy is illustrated with the challenges it poses. Then, a decentralized and coordinated control framework is proposed to regulate the output of inverter based generations and reallocate limited DER capacity for Vf control. The control framework is composed of a power regulator and a Vf regulator, which generates the supplementary signals for the primary controller. The power regulator regulates the output of grid forming inverters according to the real time capacity constraints of DERs, while the Vf regulator improves the Vf deviation by leveraging the load sensitivity to Vf. Next, the static feasibility and small signal stability of the proposed method are rigorously proven through mathematical formulation and eigenvalue analysis. Finally, a MATLAB Simulink simulation demonstrates the functionalities of the control framework. A few goals are fulfilled within the decentralized and coordinated framework, such as making the best use of limited DERs capacity, enhancing the DC side stability of inverter based generations, and reducing involuntary load shedding

Inverter PQ Control With Trajectory Tracking Capability For Microgrids Based On Physics-informed Reinforcement Learning

The increasing penetration of inverter-based resources (IBRs) calls for an advanced active and reactive power (PQ) control strategy in microgrids. To enhance the controllability and flexibility of the IBRs, this paper proposed an adaptive PQ control method with trajectory tracking capability, combining model-based analysis, physics-informed reinforcement learning (RL), and power hardware-in-the-loop (HIL) experiments. First, model-based analysis proves that there exists an adaptive proportional-integral controller with time-varying gains that can ensure any exponential PQ output trajectory of IBRs. These gains consist of a constant factor and an exponentially decaying factor, which are then obtained using a model-free deep reinforcement learning approach known as the twin delayed deeper deterministic policy gradient. With the model-based derivation, the learning space of the RL agent is narrowed down from a function space to a real space, which reduces the training complexity significantly. Finally, the proposed method is verified through numerical simulation in MATLAB-Simulink and power HIL experiments in the CURENT center.With the physics-informed learning method, exponential response time constants can be freely assigned to IBRs, and they can follow any predefined trajectory without complicated gain tuning

Virtual Synchronous Generator Control Using Twin Delayed Deep Deterministic Policy Gradient Method

This paper presents a data-driven approach that adaptively tunes the parameters of a virtual synchronous generator to achieve optimal frequency response against disturbances. In the proposed approach, the control variables, namely, the virtual moment of inertia and damping factor, are transformed into actions of a reinforcement learning agent. Different from the state-of-the-art methods, the proposed study introduces the settling time parameter as one of the observations in addition to the frequency and rate of change of frequency (RoCoF). In the reward function, preset indices are considered to simultaneously ensure bounded frequency deviation, low RoCoF, fast response, and quick settling time. To maximize the reward, this study employs the Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithm. TD3 has an exceptional capacity for learning optimal policies and is free of overestimation bias, which may lead to suboptimal policies. Finally, numerical validation in MATLAB/Simulink and real-time simulation using RTDS confirm the superiority of the proposed method over other adaptive tuning methods

Enhancing Microgrid Flexibility, Stability, and Economy with Inverter-based Resources

Microgrids face both challenges and opportunities due to the wide integration of inverter-based resources (IBRs) and advanced control techniques. On the one hand, IBRs enable the optimal use of renewable energies, which are environmentally friendly; On the other hand, IBRs introduce fast dynamics and high non-linearities to microgrids, degrading their stability and complicating the design of effective controllers. It remains an open question of how to maximize the benefits of IBRs for microgrids while overcoming the limitations. Hence, this dissertation addresses these issues by proposing new device-level control algorithms for IBRs, deriving analytical stability criteria, and integrating device-level controller design into the grid-level economic operation of microgrids. Firstly, an adaptive PQ control method with trajectory tracking capability is developed to improve the controllability and flexibility of the IBRs in microgrids, combining model-based analysis, physics-informed reinforcement learning, and power hardware-in-the-loop experiment. Exponential response time constants can be freely assigned to IBRs to follow any predefined trajectory without complicated gain tuning. Secondly, a decentralized and coordinated voltage and frequency (V-f) control framework is proposed for islanded microgrids, with full consideration of the limited capacity of distributed energy resources (DERs) and V-f dependent load. The control framework is composed of a power regulator and a V-f regulator, which generate the supplementary signals for the primary controller. Thirdly, a systematic controller design approach that ensures stability and domain of attraction (DOA) is developed for islanded microgrids. The stability conditions, i.e., certified stability, certified DOA, and their combination, are derived to rigorously guarantee whether a designated range is a subset of DOA. A systematic method for identifying the candidate control parameter set is further developed by integrating the analytical stability conditions. Lastly, the concept of virtual inertia scheduling (VIS) is proposed to efficiently handle the high penetration of IBRs. VIS is an inertia management framework targeting security-constrained and economy-oriented inertia scheduling and generation dispatch of microgrids with a large scale of DERs. It schedules the power setting points of synchronous generators and IBRs, as well as the control modes and control parameters of IBRs, to provide secure and cost-effective inertia support